System and method for determining 3d surface features and irregularities on an object

ABSTRACT

This invention provides a system and method for determining the location and characteristics of certain surface features that comprises elevated or depressed regions with respect to a smooth surrounding surface on an object. A filter acts on a range image of the scene. A filter defines an annulus or other perimeter shape around each pixel in which a best-fit surface is established. A normal to the pixel allows derivation of local displacement height. The displacement height is used to establish a height deviation image of the object, with which bumps, dents or other height-displacement features can be determined. The bump filter can be used to locate regions on a surface with minimal irregularities by mapping such irregularities to a grid and then thresholding the grid to generate a cost function. Regions with a minimal cost are acceptable candidates for application of labels and other items in which a smooth surface is desirable.

RELATED APPLICATIONS

This application is a continuation of co-pending U.S. patent applicationSer. No. 14/329,752, entitled SYSTEM AND METHOD FOR DETERMINING 3DSURFACE FEATURES AND IRREGULARITIES ON AN OBJECT, filed Jul. 11, 2014,which claims the benefit of co-pending U.S. Provisional Application Ser.No. 61/932,221, entitled SYSTEM AND METHOD FOR DETERMINING 3D SURFACEFEATURES ON AN OBJECT, filed Jan. 27, 2014, the teachings of each ofwhich applications are expressly incorporated herein by reference.

FIELD OF THE INVENTION

This invention relates to machine vision systems used in determining thedisplacement of an imaged object surface in three dimensions (3D).

BACKGROUND OF THE INVENTION

In manufacturing and assembly processes, it is often desirable toanalyze an object surface to determine the nature of features and/orirregularities. The displacement (or “profile”) of the object surfacecan be determined using a machine vision system (also termed herein“vision system”) in the form of a laser displacement sensor (also termeda laser beam “profiler”). A laser displacement sensor captures anddetermines the (three dimensional) profile of a scanned object surfaceusing a planar curtain or “fan” of a laser beam at a particular planetransverse to the beam propagation path. In a conventional arrangement,a vision system camera assembly is oriented to view the plane of thebeam from outside the plane. This arrangement captures the profile ofthe projected line (e.g. extending along the physical x-axis) on theobject surface, which, due to the baseline (i.e. the relative spacingalong the y-axis) between the beam (fan) plane and the camera causes theimaged line to appear as varying in the image y-axis direction as afunction of the physical z-axis height of the imaged point (along theimage x-axis). This deviation represents the profile of the surface.Laser displacement sensors are useful in a wide range of inspection andmanufacturing operations where the user desires to measure andcharacterize surface details of a scanned object via triangulation. Oneform of laser displacement sensor uses a vision system camera having alens assembly and image sensor (or “imager”) that can be based upon aCCD or CMOS design. The imager defines a predetermined field ofgrayscale or color-sensing pixels on an image plane that receivesfocused light from an imaged scene through a lens.

In a typical arrangement, the displacement sensor and/or object are inrelative motion (usually in the physical y-coordinate direction) so thatthe object surface is scanned by the camera, and a sequence of imagesare acquired of the laser line at desired spatial intervals—typically inassociation with an encoder or other motion measurement device (or,alternatively, at time based intervals). Each of these single profilelines is typically derived from a single acquired image. These linescollectively describe the surface of the imaged object and surroundingimaged scene and define a “range image” or “depth image”.

Other camera assemblies can also be employed to capture a 3D image(range image) of an object in a scene. For example, structured lightsystems, stereo vision systems, DLP metrology, and other arrangementscan be employed. These systems all generate an image that provides aheight value (e.g. z-coordinate) to pixels.

A 3D (range) image generated by various types of camera assemblies (orcombinations thereof) can be used to locate and determine the presenceand/or characteristics of particular features on the object surface. Achallenge is for a vision system to identify and localize such features,particularly where the surface, itself defines a varying surfaceprofile. A further challenge is to locate regions of an object that aresuitable for placement of certain items, and application of certainprocesses—such as labels and/or printing—where 3D surface irregularitiescan compromise satisfactory placement.

SUMMARY OF THE INVENTION

This invention overcomes disadvantages of the prior art by providing asystem and method for determining the location and relativecharacteristics of certain 3D surface features that generally compriseselevated or depressed regions with respect to a smooth surrounding localsurface on an overall object. The system and method employs a filterthat acts on a range (depth) image of the scene (including the object).A filter can define an annulus or other appropriate perimeter shapearound each pixel in which a best-fit surface (e.g. a best-fit plane) isestablished therefrom. The best-fit surface is used to generate a normalat the pixel. The normal is used to derive a local displacement heightwith respect to an average height of the best-fit plane. Thedisplacement height is used to establish an overall “bump” or “heightdeviation” image of the object, with which bumps, dents or otherheight-displacement/deviation features can be localized and analyzedusing appropriate vision system tools (e.g. blob tools, edge tools,caliper tools, etc.).

In an illustrative embodiment, a system for determining surface normalheight variations (features) on an object based upon a range image isprovided. A camera assembly generates a range image of the scenecontaining the object. A vision system operatively connected to, and/orassociated with the camera assembly analyzes the range image. A filter,running a filtering process that is associated with a camera sensorprocess, or that receives images from a source after acquisitionthereof, and that (a) generates a best-fit surface (for example, abest-fit plane) of average height about each of selected pixels in therange image, (b) establishes a normal to the best-fit surface,respectively for each of the pixels, and (c) determines a relativeheight displacement of a region about each of the pixels relative to thebest-fit surface. A vision system tool associated with the sensorprocess analyzes height displaced features localized by the filter.Illustratively, the filter includes a background region about each ofthe selected pixels from which the best-fit surface is established. Thefilter illustratively includes a foreground region, located withinproximate to the background region. Within the foreground region theprocess determines relative height displacement for at least one of theselected pixels. In an illustrative embodiment, at least one of a sizeand a shape of at least one of the background region and the foregroundregion is user-defined based upon at least one of a size and a shape ofthe object. Also illustratively, the camera assembly comprises adisplacement sensor that projects a line of light on the object andreceives light from the projected line at an imager in a manner thatdefines a plurality of displacement values in a height direction (e.g. alaser displacement sensor).

In another illustrative embodiment, a system and method for determiningregions of surface irregularity is provided. This system and method canbe used to locate regions on an object surface that are sufficientlyirregularity-free to enable satisfactory placement/application oflabels, printing or other items and enhancements. The system and methodemploys the above-described bump filter to locate regions on a surfacewith minimal irregularities by mapping such irregularities to a grid andthen thresholding the grid to generate a cost function. Regions with aminimal cost are acceptable candidates for application of labels andother items in which a smooth surface is desirable. In an illustrativeembodiment, a system and method for determining a region on a surfacehaving low surface irregularity is provided. The system and methodemploys a vision system tool that locates the surface and acquires a 3Dimage of at least a portion of the surface. A (bump) filter determines arelative height displacement for an area about each of a selected groupof pixels on the 3D image. In illustrative embodiments, a mappingelement can resolve the height displacement for each of a plurality ofregions on the image into a grid element with a height value. Thisheight value can be thresholded (e.g. into binary 1 and 0 values). Alocation element then cost-values the height displacement for eachregion and determines a minimal cost region that accommodates apredetermined area based directly on results provided by the bump filteror based upon a thresholded result. The thresholded result can be partof a mapped grid. This predetermined area, if it exists, can be sized toaccommodate a label, printing or other embellishment. A thresholdingelement resolves each respective height value into a binary value basedupon whether a height value is above or below at least one predeterminedparameter. The predetermined parameter is at least one of (a) heightdisplacement above or below a surrounding surface in the region and (b)area of a differing height displacement relative to the surroundingsurface in the region. Illustratively, the location element determinesif a suitable minimal cost region exists. Additionally, each gridelement can comprise part of a regular grid of predetermined density onthe image and/or the image can include at least one exclusionary zonethereon that is omitted from the minimal cost region. This exclusionaryzone can include at least one of a label, printing and an embellishmenton the surface. The surface can comprise at least part of a package, andthe minimal cost region is adapted to receive at least one of an appliedlabel and printing.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention description below refers to the accompanying drawings, ofwhich:

FIG. 1 is a schematic perspective view of a laser displacement sensorsystem acquiring an image of an object having surface normal bump/nubfeatures requiring analysis, as relative motion occurs therebetween in ascanning direction according to an illustrative embodiment;

FIG. 2 is a diagram of a range image of an exemplary object (e.g. apotato) having a non-planar surface and a plurality of bump/heightdeviation features, illustrating the difficulty of resolving suchfeatures from an unfiltered image;

FIG. 3 is a flow diagram of a filtering process for resolvingheight-displaced, surface height variations, such as bumps and dentsaccording to an illustrative embodiment;

FIG. 4 is a diagram of a range image of an exemplary object (e.g. apotato) that contains a varying surface with bumps and dents (i.e.height deviation features), showing its relation to a base plane (agrid) for use in describing the illustrative filtering process of FIG.3;

FIG. 5 is a further diagram of the exemplary range image of FIG. 4describing the establishment of foreground and background regions inaccordance with the illustrative filtering process of FIG. 3;

FIG. 6 is a further diagram of the exemplary range image of FIG. 4describing the generation of a best-fit surface (e.g. a best-fit plane)around a selected pixel in accordance with the illustrative filteringprocess of FIG. 3;

FIG. 7 is a further diagram of the exemplary range image of FIG. 4describing the generation of a distance in the normal direction withrespect to a selected pixel in accordance with the illustrativefiltering process of FIG. 3;

FIG. 8 is a further diagram of the exemplary range image of FIG. 4describing the application of the filter operator to all pixels in therange image in accordance with the illustrative filtering process ofFIG. 3 to generate an overall height deviation image

FIG. 9 is a height variation image based upon of the exemplary rangeimage of FIG. 2, based upon the illustrative filtering process of FIG. 3wherein bump features are clearly resolved for application ofappropriate analysis tools by the vision system;

FIG. 10 is a diagram of an acquired 3D range image of an exemplaryobject surface containing irregularities for which a location suitableto apply a label or other item to the surface is to be determinedaccording to an illustrative system and method;

FIG. 11 is a diagram of the acquired image of FIG. 10 in which surfaceregions thereon are derived into a grid with binary values representingflat or irregular surface features, respectively;

FIG. 12 is a flow diagram of an illustrative process for determiningpresence of one or more suitable locations/regions with sufficientflatness or surface regularity to apply an item, such as a label;

FIG. 12A is a flow diagram of an alternate process for determining thepresence of one or more suitable locations/regions with sufficientflatness or surface regularity to apply an item, using bump filterresults to construct a cost function; and

FIG. 13 is a diagram of an exemplary ball having a spherical surfacewith irregularities and exclusionary zones, showing an application forthe system and method for determining suitable locations to apply labelsand/or other items to the surface.

DETAILED DESCRIPTION I. System Overview

FIG. 1 shows a vision system arrangement 100 including a laserdisplacement sensor assembly 110 oriented to image an object 120 (alsosometimes referred to as a “part”). The displacement sensor assembly(or, simply “displacement sensor”) can be contained in a single housing112 that is mounted at an appropriate location with respect to theimaged scene. In alternate embodiments, the displacement sensor cancomprise discrete, separated subcomponents. The object 120 and thedisplacement sensor 110 are in relative motion (double arrow MY) witheither the displacement sensor 110, the object 120, or both, moving (thescan motion direction) along at least one axis of the relativecoordinate system 124 (in this example, the physical y-axis direction).In a typical arrangement, the object 120 is located on a precisionmotion stage 130 (shown in phantom) that provides motion data from anencoder or similar device operatively connected to the motion stagedrive system 132 in a manner clear to those of skill in the art.

Note that the above-described displacement sensor 110 is one of avariety of imaging devices and/or camera assemblies that can be used togenerate a range image for use in accordance with the system and methodherein. By way of non-limiting example, a camera assembly as definedherein can include a structured light system, a stereo vision system,DLP metrology and/or combinations thereof, among other types ofarrangements. Also, as described further below, the system and methodcan be employed on images that are acquired and then stored for furtherprocessing. Such images can be transmitted for processing by the systemand method (i.e. a filtering process and associated vision system tools)in a manner in which the processor is free of connection to the cameraassembly or other image acquisition arrangement.

The object 120 is, by way of example, a structure having a surface 140that defines a generally non-planar shape, and by way of non-limitingexample, more particularly defines a somewhat twisted surface in whichthe height (i.e. the displacement dimension along the physical z axis)varies with respect to the x-axis and y-axis directions. Notably, theobject surface 140 includes a plurality of bumps or “nubs” 142 that eachvary in displacement along the physical z-axis direction—typically beingdirected in a “positive” direction away from the object surface. Thepositive displacement can be characterized as residing along a directionand distance that is normal to the object surface (i.e. surface-normalto a plane that is tangent to the surface at a point on the nub). Thesenubs 142, thus define a height displacement with respect to thesurrounding surface. This is only one example of a wide range ofpossible surface shapes, and is described as a useful example of theprinciples of the illustrative embodiment of the system and method. Infurther examples the outer, surrounding surface can define a morecomplex perimeter with additional features.

The displacement sensor assembly 110 includes a laser assembly 150 thatincludes a lens or other optical arrangement 152 (e.g. a Powell lens ora cylindrical lens) that projects a curtain or fan 154 of laser light,toward the object 120 in a manner that forms a line 156 along thephysical x-axis (transverse to the scan motion direction MY). The fan154 resides generally in a plane that, in this embodiment, isapproximately (or substantially) perpendicular to a reference plane,such as the physical x-y plane of the motion stage 130 and/or a plane ofthe surrounding object surface 140. The displacement sensor 110 alsoincludes a camera assembly (dashed box 160) consisting generally of alens/optics assembly 162 potentially arranged in a well-knownScheimpflug configuration and an imager assembly (“imager”) 164. Theimager 164 contains an array of photosensitive pixels at a predeterminedresolution, and the camera assembly generally defines an optical axis OAthat intersects the object 120 in the region of the line 156. Theoptical axis OA defines a non-parallel (e.g. acute) angle θ with respectto the plane (and associated line 170) of the laser fan 154. Thus, anyphysical z-axis height variation/displacement in the line along thephysical x-axis is imaged by the camera assembly 160 as a variation inposition (along the image axis YI) in the resulting line 171 of atwo-dimensional (2D) image 172 as shown generally in the exemplary (andoptional) display 174.

Note that the arrangement of axes (x, y, z) is a convention and otherrepresentations of relative orientation (e.g. polar coordinates) areexpressly contemplated. As shown, rotation about a given axis is alsorepresented, for example by the double-curved arrow Rx, which depictsrotation about the physical x-axis.

The imager 164 is operatively interconnected to a vision systemprocessor 180 that is contained within the housing 112 of thedisplacement sensor 110 in this embodiment. In alternate embodiments,some or all of the vision system processor components and/or processescan reside remote from the housing (on an interconnected PC, forexample). The vision system processor 180 performs a displacement sensorprocess 182 in accordance with the illustrative embodiment described indetail further below. The processor 180 also receives relative motionsignals from the motion stage 130 or another moving device (e.g. agantry or manipulator holding the displacement sensor and/or theobject). Images are acquired in association with such motion signals sothat the line 156 is registered with the physical position in the scene.

As also described below, the image data (i.e. a 2D range image) for theobject is transmitted to downstream data handling devices and processesfor use in various production activities, such as inspection, qualitycontrol, object manipulation, assembly, etc.

Users typically desire measurements characterized in physical units(e.g., millimeters or microns in x and y and z), rather than image units(i.e. pixels/pels in x, encoder motion-based triggers in y, and rowdisplacements in z). For this reason, displacement sensors are typicallycalibrated to convert between image pixel outputs and physical units,such that the calibrated displacement sensor output is in physicalunits. Displacement sensor calibration can be performed by themanufacturer such that the displacement sensor provides output inphysical units (such as the model DS1100 available from CognexCorporation of Natick, Mass.). Alternatively, sensors can provide outputin uncalibrated image units. When the sensor output is in uncalibratedimage units, the user or application is tasked with handling input datain image pixel units.

Note also, as used herein the terms “process” and/or “processor” shouldbe taken broadly to include a variety of electronic hardware and/orsoftware based functions and components. Moreover, a depicted process orprocessor can be combined with other processes and/or processors ordivided into various sub-processes or processors. Such sub-processesand/or sub—processors can be variously combined according to embodimentsherein. Likewise, it is expressly contemplated that any function,process and/or processor here herein can be implemented using electronichardware, software consisting of a non-transitory computer-readablemedium of program instructions, or a combination of hardware andsoftware.

II. Filter and Sensor Process

Reference is made briefly to FIG. 2, which shows a range image 200 of anobject, which in this example is a potato with a varying surfacegeometry and a variety of height deviating/height-displaced features210. These height-displaced features 210 (“bumps” or “nubs” are imagedand appear in a differing shade and/or color. However, it should beclear that such “bumps” are difficult to resolve due to the overallvariation in surface height/geometry for the object. The geometry of theobject surface (e.g. a twisted shape) renders identification ofindividual nubs (or other surface-normal height varying features—such asdents) difficult to identify. This can slow or render ineffective theoverall inspection process using a vision system. Therefore, in order toenhance the effectiveness and accuracy of the vision system's inspectionprocess, a filtering process 190 (FIG. 1) in accordance with anillustrative embodiment is applied as part of the overall sensor process182. The filtered image is described further below with reference toFIG. 9.

As shown in FIG. 1, images can alternatively (or additionally) beprovided from a source 188 that stores image data for furtherprocessing. The image processor that includes the filtering process 190and/or vision system tools 183 can be separate from the cameraassembly's functions and can be located in a separate device, or canreside within the camera assembly (110) as shown. Thus, it is expresslycontemplated that the system and method can operate on images free ofinterconnection with the camera assembly at the time filtering processesare performed.

The filtering process 190 is described with further reference to theflow diagram of FIG. 3. The process begins when an object within thescene is imaged to generate a range image (step 310). With referencealso to FIG. 4, a range image 400 exemplary object 410 having a variablesurface geometry (e.g. a potato) is shown. The depicted exemplary objectsurface defines a plurality of positive-height features (e.g. bumps ornubs) and a plurality of negative-height features (e.g. dents). Therange image processed by the illustrative filter defines a range (z-axisdepth) image of the object (potato), wherein each image pixel representsthe height (e.g., in mm) of the object above a base plane (exemplarygrid 420 and surface 130 (FIG. 1)) on which the object rests. Thecoordinates of two exemplary z pixels 430, 432 can be represented asZ₀(X₀, Y₀) and Z₁(X1, Y1), respectively. The full range image definesthe height z at each pixel location (xi, yj) of the image.

In step 320, the filtering process 190 first generates a surface heightand a surface normal direction (using a plane-fit tool). The process 190then computes a height deviation measurement in the normal direction(using a height tool). The plane fit creates the first two measurementsand the height tool creates the third measurement. Referring also to theimage 400 of Fig, 5, the process 190 computes an appropriate foregroundregion and background region (or plurality of such regions) for eachpixel (cross 510). The exemplary pixel (cross 510) is shown with abackground region (annulus 520) centered around it at a radial distancefrom a foreground surface (circle 530), which represents the normalheight region. The bump height can be computed from each pixel in theimage or a subset of pixel that can be determined by various processes.Note that a plurality of background and/or foreground regions can beestablished for a given pixel. The two regions are used to compute thelocal “bump” height of the pixel (cross 510) as described further below.The size, shape and number of such regions can be determined by the uservia an appropriate user interface (for example an interface associatedwith display 174 in FIG. 1). These regions can be established based upongeneral knowledge about the size, shape and/or location of the bumps (orother surface normal height features) on the object. In this manner, theuser sets the parameters of the filter so as to optimize determinationof such features on the object.

The process 190 next generates (for each of one or more (a plurality) ofregions (e.g. pixel 510)) in the image 400 a best-fit surface andsurface normal for z pixels in that region (step 330). Note that thebest-fit surface is typically a plane, but it is expressly contemplatedthat the best-fit surface can comprise all or a portion of a sphere, orother predefined 2D or 3D shape. Such shape is selected and defined,potentially by a user input, to provide the best results for aparticular height deviation feature. The illustrative best-fitsurface/plane 610 is generated from the background region (annulus 520),and represents a local background estimate of the “smooth” objectsurface in the area of the pixel 510 based upon a height profile forpixels in the background region. The surface normal 620 extends from thepixel 510 and is perpendicular to the illustrative plane 610, pointingin a positive height direction. Note that the regions used to generatethe best-fit surface and/or the bump height (background and foreground)need not be contiguous when pluralities of such regions are employed.This can allow the system to omit certain features that fall outside the“smooth” or “bump” geometry that characterizes the rest of the surfaceon the object.

In step 340, and with reference to FIG. 7, the process 190 then computesthe relative distance of the foreground region 530 for the pixel 510from the best-fit plane 610 (FIG. 6). The foreground region 530 definesthe set of z pixels whose average, signed distance is computed from theestimated local background plane 610. Note that the underlying baseplane (420, FIG. 4) can be computed in a variety of ways. The positionof the scan line in the field of view in portions of the imaged scenethat represent a minimum height can be associated with a base plane(z-axis) height. This average distance can optionally be obtained bycomputing the center of mass (CM) of the foreground points and thencomputing the (signed) distance of this CM point to the estimated localbackground surface/plane. Note that the distance is signed positive ifthe foreground region is above the surface/plane 610 (on the same sideof the plane as the outward surface normal shown in the previous slide),and signed negative if it is below the plane 610.

In step 350, and with reference to FIG. 8, the image 400 is analyzed ateach pixel using the filter operator described above (with one or morebackground regions and one or more foreground regions per pixel), togenerate an overall “bump image”, or more generally a “height deviationimage” result. That is, the process repeats steps 330 and 340 above forall (or a selected subset of all) pixels in the range image. The filteroperator 810, 820, 830, and 840 is shown at various locations around theimage 400. Note that negative-height “dents” can also be analyzed aswell as positive-height “bumps” 870. Areas on the base plane (420 inFIG. 4) are shown as spots 850. The resulting height deviation image isa signed image that represents the local height deviations from asmoothed surface representation of the object (e.g. a potato). Localdents (e.g. dent 860) in the surface define negative heights and localbumps (e.g. bump 870) on the surface define positive heights, relativeto the local surrounding surface.

Based upon the height deviation “bump” image derived in step 350, thesensor process can apply various vision system tools (183 in FIG. 1)including, but not limited to, edge-detection tools, blob analysis toolsand/or caliper tools to analyze such bump, dent or other features forconformance with desired specifications (step 360). Desirably, suchbumps and dents are localized by the filter so that the vision systemcan direct analysis to these features more efficiently and with reducedprocessing time.

IV. Determination of Irregularity-Free Regions on an Object Surface

A particular problem for which 3D vision systems described above (i.e.laser profiling systems) can be employed relates to the location andplacement of labels, printing and/or other enhancements on an objectsurface that contains irregularities. In one example, the surfacecomprises a side of a box, carton or container that can include tape orother applied items that create a rough surface and/or defineexclusionary regions to which no further printing or labels should beapplied. Reference is made to FIG. 10, which shows a typical objectsurface 1010 containing regions with surface irregularities andexclusionary zones. By way of example, the surface 1010 defines atypical box surface to which a label 1040 with predetermined dimensions(e.g. length LL and width WL) should be applied. It is desirable thatthe label 1040 be placed to avoid a shipping barcode (or otherinformation) 1020 that should be visible during various stages of theshipping and/or logistics process. It is also desirable that the labelbe placed to avoid overlap onto a sealing tape strip 1030, which caninclude an irregular surface with wrinkles and troughs 1032. That is,the label may not adhere properly to this region, and/or will appearsloppily placed. Stated in the alternative, it is desired that the labelbe placed in a region that is relatively smooth and generally free of 3Dsurface irregularities.

In order to allow automated placement of the exemplary label 1040 ontothe box surface 1010 in a rapid and reliable manner, where the boxsurface can vary in size, shape and layout, making predictable placementof a label challenging for the system. Likewise, the box can bepresented to the system in a range of orientations and/or tilt angles,which also renders positioning of the label challenging. As shown inFIG. 10, the box surface 1010 is angled at angle A with respect to thedirection of line motion along, for example, a conveyor.

Reference is now made to FIG. 11, which shows an acquired 3D image 1100of the box surface 1010. The image 1100 includes the barcode feature1120 and the tape strip feature 1130, which each represent exclusionaryregions for placement of the label 1040. Briefly, the system and methoddetermines a series of grid locations 1150 within the overall image,once the surface has been located. These grid locations can be regularlyspaced as shown or irregularly spaced—for example, oriented relative toknown regions of interest on the surface. The nature of the 3D featuresat each grid location is determined. Those grid locations with surfaceirregularities, (e.g. bumps or dips that exceed a predeterminedheight/depth threshold) are flagged as shown by the applied X marks(e.g. grid locations 1152). As described below, the grid forms the basisof the decision-making process on whether a given region is appropriateto accommodate a label or other embellishment—or if no region isavailable on the surface to accommodate the item.

With further reference to FIG. 12, a location process 1200 is shown anddescribed in further detail. In step 1210, the process 1200 (typicallycarried out by the vision system processor 180 (FIG. 1) using thelocation determination process 191) acquires a 3D range image (i.e. aprofile) of a surface on an object of interest using any appropriate 3Dimaging device including, but not limited to, the vision systemarrangement 100 in FIG. 1. The object can be passed through the visionsystem arrangement and tracked using, for example, an encoder thatmeasures travel of a conveyer that transports the object through theimaged scene. The 3D pixel data acquired by the vision system camera isused to provide an image data set for use in various follow-on visionsystem processes. Illustratively, in step 1220, the acquired image isused (for example in 2D) to locate the edges and corners (if applicable)of the surface, and/or any features of interest (e.g. seams, labels,barcodes, fiducials, etc.) on the surface. A variety of conventionaland/or custom vision system tools can be used in this step—for example,edge finders, blob tools, contrast tools, and other tools andapplications that are clear to those of ordinary skill (e.g.tools/applications available from Cognex Corporation of Natick, Mass.).

In process step 1230, the above-described bump filter operates on theimage data set to determine locations/regions of surface irregularities(i.e. 3D bumps and dips) on the surface image. Then in step 1240, theprocess 1200 generates a grid map with a predetermined granularity(which can be defined finer or coarser based upon the user's needs)across some or all of the surface image. An exemplary grid is shown byelements 1150 in FIG. 11. The grid is thresholded, based uponsystem-based parameters, or user-based parameters (using, for example anappropriate graphical user interface (not shown)). The thresholding cancreate a binary (1-bit) pattern of 1s and 0s that are values based uponwhether a particular grid element contains a surface irregularity ofsufficient magnitude (e.g. z-axis height and/or x-y-axis size) to meetthe minimum threshold for a surface irregularity that is consideredproblematic by the system in the application of a label. As describedabove, the spacing and layout of the grid can be regular in the x andy-axis directions or it can be arranged according to another pattern ofeither regular or irregular spacing. For example, the spacing along thex-axis direction can be less than that along the y-axis direction. Also,certain regions of the image can specify closer grid spacing in either(or both) direction(s) than other regions—for example where region iscrowded with small features that should be avoided.

In step 1250, which can be optional, the system can generate maskswithin the grid representative of exclusionary zones (i.e. forbiddenzones) around regions of the image that contain certain features thatshould not be covered with labels or other items. For example, region1120 in the image of FIG. 11 includes a barcode that should be leftuncovered and clearly exposed on the exemplary package. Theseexclusionary zones can be predefined where the feature is in a known andpredictable location on the package. Alternatively, the exclusionaryzone can be defined using a vision system tool—for example, an IDcode-finding tool. The size of the exclusionary zone is sufficient toensure that the feature remains unaffected by the application of a labelor other item. Note that exclusionary zones can also be defined basedupon locations on the image where further process will occur—such asapplication of additional labels, postage, stamping, etc. Such locationsare predefined (pre-trained) in the process, and are typically notgenerated based upon the determination of a feature in the image.

In step 1260, the process 1200 applies a cost function to thethresholded grid on an iterative basis. This cost function computesregions with a high value due to surface irregularity (and/or optionallyexclusionary zones due to printing, etc.). In generating such values,the process 1200 counts the number of 1s (e.g. grid elements shown withan X in FIG. 11 (1152)) versus 0s (1150) for each grid location in theimage over the range of the label dimensions (e.g. x and y length andwidth). In step 1270, the process then locates the region where thetotal value is minimized (e.g. a global minimum with the least number ofXs). As shown in the example of FIG. 11, the region 1170 of the tapestrip is dense with high value grid elements (Xs). Likewise, the bottomarea 1180 of the image contains several high value grid elements.Conversely, the right top of the image 1100 is relativelyirregularity-free (i.e. the flattest print-free region) and thereforeprovides a desirable location form applying the exemplary label as shownby the dashed box 1160.

Note that the cost function shown and described herein is illustrativeof a wide range of techniques, mechanisms and/or approaches to mappingthe image. Alternatively, the cost function can be constructed directlyon the results of the bump filter from step 1230. As such, the modifiedflow diagram shown in FIG. 12A applies. Note that similar steps to thoseshown and described for FIG. 12 are provided with like referencenumbers, and the above description thereof applies. In particular, step1240 is omitted from FIG. 12A and the bump filter in step 1230 generatesregions with surface irregularity that directly indicate locations inwhich a bump or dip resides on the overall surface. The locations areprocessed in step 1292 to generate a cost function free of any grid map.That is, the cost function uses height (and coverage area) informationfor bumps and dips to determine whether a particular region is suitablefor the item/label to-be-applied to the surface. The optionalexclusionary zones can also be processed in step 1292 to exclude regionswhere a label or other forbidden feature resides

Once the optimal location for label (or other item) application isdetermined in either FIG. 12 or 12A, the process can trigger otherdevices or systems to apply the label/item in step 1280 (shown inphantom) to that location. This can entail servoing a robot manipulatorand/or a motion stage containing the object so that the label/item isapplied to the desired location on the surface. Alternatively, step 1280can indicate that no suitable location exists to apply a label or otheritem to the surface due to excess irregularity or size constraints ofthe available flat surface. Where no suitable location is available, theobject can be flagged by an alarm and/or rerouted from the conveyor line(for example) to a special handling location.

It is expressly contemplated that the user interface or otherprogramming mechanism can be used to adjust the sensitivity of theprocess by altering or tuning the bump filter masks to identifyirregularities (non-flat features) of a certain height and/or size. Thegrid density/granularity can also be adjusted to change the overallsensitivity of the process.

With reference to the object 1300 of FIG. 13 (e.g. a soccer ball withalternating white and black pentagonal shapes 1310 and 1320,respectively) the process 1200 has the ability to accommodatelow-frequency variation of the surface, such bas the continuouscurvature of the spherical ball. The process 1200 can identify anappropriate location (dashed box 1330) for an applied item, whilelocating ridges 1340 (denoted by grid Xs) that should be avoided. Theblack pentagons 1320 can also be avoided as locations for the item byestablishing exclusionary zones as described above in step 1250. Theillustrative process 1200 effectively locates surfaces with lowirregularity even on a sloped or curved surface, such as the depictedball. Conversely other algorithms, such as a conventional fit planealgorithm, would generally fail on the surface of the depicted ball.

The system and method is also advantageous over an alternative techniquein which a plane is fit to a specific area and then this area is grownas long as plane's fit quality remains acceptable (up to a size thatoccupies the label). In such an alternative technique, the processingoverhead required is often substantially greater as the system iteratesthrough successive area-growth operations.

Referring again to the exemplary range image 200 of FIG. 2, subsequentto application of the filtering process of FIG. 3, the height deviationimage can be represented by the diagram of FIG. 9. The object 900 isshown with a series of resolved spots 910 that clearly differentiate theheight deviating features (bumps) from the surrounding surface (due totheir depicted sharp contrast with the surroundings). As such, thevision system can direct further analysis specifically to thesehighlighted regions with greater accuracy and efficiency.

Thus, it should be clear that the illustrative surface normal bumpfilter described above is particularly desirable for use in analyzingobjects having non-flat, but smooth surfaces, with local dents andbumps. For a simple surface such as a plane with dents, the illustrativebump filter would give comparable results as those obtained by fitting asingle plane, then measuring the distance of each point (pixel) on thesurface from the fit plane. However, because the bump filter estimateslocal planes, it is able to accommodate a wide range of non-flatsurfaces with variable and a priori unknown surface geometry. Notably,because the illustrative bump filter measures distance normal to thesurface, it produces nearly identical height estimates at correspondingpoints even when a 3D object is rotated in depth and reimaged with arange sensor. This invariance greatly simplifies subsequentprocessing/analysis of the bump height image in order to detect surfacedefects by various vision system tools.

It is expressly contemplated that particular enhancements to theabove-described process(es) can be employed in other applications.Illustratively, the process can employ the residual error of the planefit to qualify the height measurement. Where an annulus is (for example)used for the background and a patch in the foreground for the height,the process can both respond to the (bump) feature being sought and alsocan respond all along a particular linear feature (e.g. a ridge). Theplane fit can exhibit high residual fit error at the points where theannulus is centered on the ridge, and that response can be used toreject the height measurements at those points.

In the determination of sufficiently flat and/or regular surfaces forapplication of labels or other items, the foregoing embodiment providesan efficient and variable system and method. This system and methodoperates on surfaces with both high-frequency and low-frequency surfacevariation, and effectively determines appropriate locations forapplication of items. These items can include applied labels, printing,etching, decals, and/or any other appropriate embellishments to thesurface that desirably reside within a relatively regular region.Likewise the underlying surface can be flat, curved or partially flatand partially curved. A variety of soft and/or hard surface materialsand/or finishes are also contemplated including specular, opaque,translucent, transparent and combinations thereof. Moreover, through useof masks and/or exclusionary zones, creases or other knownirregularities can be overlooked in determining a location for itemapplication. In such a manner an item can be placed across a crease,fold, crown edge or other known feature.

The foregoing has been a detailed description of illustrativeembodiments of the invention. Various modifications and additions can bemade without departing from the spirit and scope of this invention.Features of each of the various embodiments described above may becombined with features of other described embodiments as appropriate inorder to provide a multiplicity of feature combinations in associatednew embodiments. Furthermore, while the foregoing describes a number ofseparate embodiments of the apparatus and method of the presentinvention, what has been described herein is merely illustrative of theapplication of the principles of the present invention. For example, asused herein various directional and orientational terms such as“vertical”, “horizontal”, “up”, “down”, “bottom”, “top”, “side”,“front”, “rear”, “left”, “right”, and the like, are used only asrelative conventions and not as absolute orientations with respect to afixed coordinate system, such as gravity. Also, height (z-axis) accuracycan be improved by setting camera assembly exposure so that detectedfeatures in the image remain unsaturated. More particularly, the rangeimage generated herein can be provided by a variety of 3D imagingsystems including, but not limited to a LIDAR-equipped camera, astereoscopic camera assembly, or combinations of such arrangements. Asdescribed generally above, the foreground and background regions can bespecified with any size and shape that meets the requirements of a givenapplication. Also since, the most processing-intensive aspect of thesurface normal bump filter is typically the local plane-fittingoperation, this task can be expedited by recursively updating theplane-fit regression matrix as the background region is slid across theimage. Also as noted above, the local plane fitting operation can avoiduse of all points in the background region—and instead can employ arobust plane-fitting procedure (e.g. RANSAC) to exclude outlier pointsfrom the fit. Illustratively, invariance of measurements to the objectsurface orientation can also be further improved by varying the shape ofbackground and foreground regions according to the local surface normaldirection. This would allow presentation of foreground and backgroundregions whose shape is constant in the surface coordinates of the 3Dobject (rather than being constant in the x-y space of the range image).Accordingly, this description is meant to be taken only by way ofexample, and not to otherwise limit the scope of this invention.

What is claimed is:
 1. A system for determining surface heightvariations on an object based upon a range image comprising. a cameraassembly that generates a range image; a vision system that analyzes therange image; a filter that (a) generates a best-fit surface about eachof selected pixels in the range image, (b) establishes a normal to thebest-fit surface, respectively for each of the pixels, and (c)determines a relative height displacement of a region about each of thepixels relative to the best-fit surface; and a vision system tool thatanalyzes height displaced features localized by the filter.
 2. Thesystem as set forth in claim 1 wherein the filter includes a backgroundregion about each of the selected pixels from which the best-fit surfaceis established.
 3. The system as set forth in claim 2 wherein the filterincludes a foreground region, located proximate to the background regionand including at least one of the selected pixels in which the relativeheight displacement is determined.
 4. The system as set forth in claim 3wherein at least one of a size and a shape of at least one of thebackground region and the foreground region is user-defined based uponat least one of a size and a shape of the object.
 5. The system as setforth in 1 wherein the camera assembly comprises a displacement sensorthat projects a line of light on the object and receives light from theprojected line at an imager in a manner that defines a plurality ofdisplacement values in a height direction.
 6. The system as set forth inclaim 1 wherein the best-fit surface comprises a best-fit plane.
 7. Amethod for determining surface height variations on an object based upona range image comprising the steps of: (a) generating a best-fit surfaceof average height about each of selected pixels in the range image; (b)establishing a normal to the best-fit surface, respectively for each ofthe pixels; (c) determining a relative height displacement of a regionabout each of the pixels relative to the best-fit surface; and (d)analyzing, with a vision system tool, localized height displacedfeatures.
 8. A system for determining a region on a surface having lowsurface irregularity comprising: a vision system tool that locates thesurface and acquires a 3D image of at least a portion of the surface; afilter that determines a relative height displacement for an area abouteach of a selected group of pixels on the 3D image; and a locationelement that cost-values the height displacement for each region basedupon results provided by the filter and that determines a minimal costregion that accommodates a predetermined area.
 9. The system as setforth in claim 8 wherein the predetermined area is sized to accommodatea label, printing or other embellishment.
 10. The system as set forth inclaim 8 further comprising a mapping element that resolves the heightdisplacement for each of a plurality of regions on the image into a gridelement with a height value.
 11. The system as set forth in claim 8further comprising a thresholding element that resolves each heightvalue respectively into a binary value based upon whether a height valueis above or below at least one predetermined parameter.
 12. The systemas set forth in claim 11 wherein the predetermined parameter is at leastone of (a) height displacement above or below a surrounding surface inthe region and (b) area of a differing height displacement relative tothe surrounding surface in the region.
 13. The system as set forth inclaim 8 wherein the filter (a) generates a best-fit surface of averageheight about each of selected pixels in the range image, (b) establishesa normal to the best-fit surface, respectively for each of the pixels,to (c) determine the height displacement of the about each of the pixelsrelative to the best-fit surface.
 14. The system as set forth in claim13 wherein the filter includes a background region about each of theselected pixels from which the best-fit surface is established.
 15. Thesystem as set forth in claim 14 wherein the filter includes a foregroundregion, located proximate to the background region and including atleast one of the selected pixels in which the relative heightdisplacement is determined.
 16. The system as set forth in claim 15wherein at least one of a size and a shape of at least one of thebackground region and the foreground region is user-defined based uponat least one of a size and a shape of the object.
 17. The system as setforth in 8 wherein the camera assembly comprises a displacement sensorthat projects a line of light on the object and receives light from theprojected line at an imager in a manner that defines a plurality ofdisplacement values in a height direction.
 18. The system as set forthin claim 8 wherein the location element determines if a suitable minimalcost region exists.
 19. The system as set forth in claim 8 wherein eachgrid element comprises part of a regular grid of predetermined densityon the image.
 20. The system as set forth in claim 8 wherein the imageincludes at least one exclusionary zone thereon that is omitted from theminimal cost region.